{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Developing, Training, and Deploying a TensorFlow model on Google Cloud Platform (using CloudShell and Cloud AI Platform)\n",
    "\n",
    "In this notebook, we will develop a Keras model to predict flight delays using TensorFlow 2.0 as the backend. Unlike [flights_model_tf2.ipynb](flights_model_tf2.ipynb), we will use bash commands so that these can be run from CloudShell. We will also deploy to Cloud AI Platform so that the model can be executed in a serverless way."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Install necessary Python package"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%pip install --upgrade --quiet cloudml-hypertune"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Restart the kernel after doing the pip install"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Set environment variables\n",
    "\n",
    "I'm doing this for the notebook. In CloudShell, you'd do:\n",
    "<pre>\n",
    "export PROJECT=cloud-training-demos\n",
    "</pre>\n",
    "etc."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "parameters"
    ]
   },
   "outputs": [],
   "source": [
    "# change these to try this notebook out\n",
    "BUCKET = 'cloud-training-demos-ml'\n",
    "PROJECT = 'cloud-training-demos'\n",
    "REGION = 'us-central1'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "os.environ['BUCKET'] = BUCKET\n",
    "os.environ['PROJECT'] = PROJECT\n",
    "os.environ['REGION'] = REGION"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "gcloud config set project $PROJECT\n",
    "gcloud config set compute/region $REGION"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Try out different functions in the model\n",
    "\n",
    "Reading lines"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "export PYTHONPATH=\"$PWD/flights\"\n",
    "python3 -m trainer.task --bucket $BUCKET --train_batch_size=3 --func=read_lines"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finding average label"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "export PYTHONPATH=\"$PWD/flights\"\n",
    "python3 -m trainer.task --bucket $BUCKET --num_examples=100 --func=find_average_label"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Linear model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "export PYTHONPATH=\"$PWD/flights\"\n",
    "gcloud storage rm --recursive --continue-on-error gs://$BUCKET/flights/trained_model\n",
    "python3 -m trainer.task --bucket $BUCKET --num_examples=1000 --func=linear\n",
    "\n",
    "gcloud storage ls gs://$BUCKET/flights/trained_model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Wide-and-deep model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "export PYTHONPATH=\"$PWD/flights\"\n",
    "gcloud storage rm --recursive --continue-on-error gs://$BUCKET/flights/trained_model\n",
    "python3 -m trainer.task --bucket $BUCKET --num_examples=1000 --func=wide_deep\n",
    "\n",
    "gcloud storage ls gs://$BUCKET/flights/trained_model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "model_dir=$(gcloud storage ls gs://$BUCKET/flights/trained_model/export | tail -1)\n",
    "echo $model_dir\n",
    "saved_model_cli show --tag_set serve --signature_def serving_default --dir $model_dir"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Run on full dataset\n",
    "\n",
    "Submit the Python package to Cloud AI Platform Training and have it process full dataset.\n",
    "This is serverless, so you can do the equivalent by running ./train_model.sh from CloudShell.\n",
    "You don't need a Notebook environment to do this."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "JOBID=flights_$(date +%Y%m%d_%H%M%S)\n",
    "gcloud storage rm --recursive --continue-on-error gs://$BUCKET/flights/trained_model\n",
    "\n",
    "gcloud beta ai-platform jobs submit training $JOBID \\\n",
    "   --staging-bucket=gs://$BUCKET  --region=$REGION \\\n",
    "\n",
    "   --master-machine-type=n1-standard-4 --scale-tier=CUSTOM \\\n",
    "   -- \\\n",
    "   --bucket=$BUCKET --num_examples=1000000 --func=wide_deep"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The final validation RMSE was 0.214"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "model_dir=$(gcloud storage ls gs://$BUCKET/flights/trained_model/export | tail -1)\n",
    "echo $model_dir\n",
    "saved_model_cli show --tag_set serve --signature_def serving_default --dir $model_dir"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Option 1: Single model training on Cloud AI Platform\n",
    "This will take 15-20 minutes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "./train_model.sh $BUCKET wide_deep 500000"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Option 2: Hyperparameter tuning on Cloud AI Platform\n",
    "This will take 2-3 hours"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "\n",
    "Comment this line. Make sure to wait for hyperparam job to finish, find best model and change BEST_MODEL in following cell!\n",
    "\n",
    "JOBID=flights_$(date +%Y%m%d_%H%M%S)\n",
    "gcloud storage rm --recursive --continue-on-error gs://$BUCKET/flights/trained_model\n",
    "\n",
    "gcloud ai-platform jobs submit training $JOBID \\\n",
    "   --staging-bucket=gs://$BUCKET  --region=$REGION \\\n",
    "   --module-name=trainer.task \\\n",
    "   --python-version=3.7 --runtime-version=2.1 \\\n",
    "   --package-path=${PWD}/flights/trainer \\\n",
    "   --config=hyperparam.yaml \\\n",
    "   -- \\\n",
    "   --bucket=$BUCKET --num_examples=100000 --func=wide_deep"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Deploy model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "MODEL_NAME=flights\n",
    "VERSION_NAME=tf2\n",
    "BEST_MODEL=\"\"      # Option 1: No hyperparameter tuning\n",
    "#BEST_MODEL=\"15/\"   # Option 2: CHANGE AS NEEDED \n",
    "EXPORT_PATH=$(gcloud storage ls gs://$BUCKET/flights/trained_model/${BEST_MODEL}export | tail -1)\n",
    "echo $EXPORT_PATH\n",
    "\n",
    "if [[ $(gcloud ai-platform models list --format='value(name)' | grep $MODEL_NAME) ]]; then\n",
    "    echo \"$MODEL_NAME already exists\"\n",
    "else\n",
    "    # create model\n",
    "    echo \"Creating $MODEL_NAME\"\n",
    "    gcloud ai-platform models create --regions=$REGION $MODEL_NAME\n",
    "fi\n",
    "\n",
    "if [[ $(gcloud ai-platform versions list --model $MODEL_NAME --format='value(name)' | grep $VERSION_NAME) ]]; then\n",
    "    echo \"Deleting already existing $MODEL_NAME:$VERSION_NAME ... \"\n",
    "    gcloud ai-platform versions delete --quiet --model=$MODEL_NAME $VERSION_NAME\n",
    "    echo \"Please run this cell again if you don't see a Creating message ... \"\n",
    "    sleep 10\n",
    "fi\n",
    "\n",
    "# create model\n",
    "echo \"Creating $MODEL_NAME:$VERSION_NAME\"\n",
    "gcloud ai-platform versions create --model=$MODEL_NAME $VERSION_NAME --async \\\n",
    "       --framework=tensorflow --python-version=3.7 --runtime-version=2.1 \\\n",
    "       --origin=$EXPORT_PATH --staging-bucket=gs://$BUCKET"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%writefile example_input.json\n",
    "{\"dep_delay\": 14.0, \"taxiout\": 13.0, \"distance\": 319.0, \"avg_dep_delay\": 25.863039, \"avg_arr_delay\": 27.0, \"carrier\": \"WN\", \"dep_lat\": 32.84722, \"dep_lon\": -96.85167, \"arr_lat\": 31.9425, \"arr_lon\": -102.20194, \"origin\": \"DAL\", \"dest\": \"MAF\"}\n",
    "{\"dep_delay\": -9.0, \"taxiout\": 21.0, \"distance\": 301.0, \"avg_dep_delay\": 41.050808, \"avg_arr_delay\": -7.0, \"carrier\": \"EV\", \"dep_lat\": 29.984444, \"dep_lon\": -95.34139, \"arr_lat\": 27.544167, \"arr_lon\": -99.46167, \"origin\": \"IAH\", \"dest\": \"LRD\"}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!gcloud ai-platform predict --model=flights --version=tf2 --json-instances=example_input.json"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "./call_predict.py --project=$PROJECT\n",
    "echo\n",
    "echo \"With reasoning\"\n",
    "./call_predict_reason.py --project=$PROJECT"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Copyright 2016-2020 Google Inc. Licensed under the Apache License, Version 2.0 (the \"License\"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Tags",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
